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introduction 


• Today’s Sci/Ops Outlook: 

- Science Budgets are being 
slashed, very little money 
is allocated for Science 
Operations activities 

- At Goddard, operations space is extremely limited, 
expensive, and mainly reserved for flight operations 
(who tend to have deeper pockets) 

• It was mid-2010, and the LADEE Science Operations 
Center was looking for a home 
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Scbnos Operations Challenges 


We discovered many Goddard science payloads ran their ops out of 
someone’s cube or office! 

• The LADEE SOC would need to be a true ops facility, with advanced 
requirements such as: 

- Real-time remote commanding of payloads 

- Automated monitoring and alert 

- Automated real-time and offline data distribution 

- 24/7 keycard-restricted secure access 

- Battery and Generator-backed power distribution 

- Access to the NASA Secure Mission Networks (data and voice) 

• We realized quickly-these reqs. couldn’t be met by a server or two 
living in someone’s cube! 
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Friends in Low 


• For many of the options open to us, the 
cost-of-entry for facility build-out was just 
too high for LADEE to foot the bill alone. 

• Luckily we were able to make some 
friends with a similar set of needs and 
timetable to our own! 

- LADEE Neutral Mass Spectrometer 
(NMS) Instrument Ops 

- MSL Sample Analysis at Mars (SAM) Payload Ops 

- MAVEN Neutral Gas Neutral Gas and Ion Mass Spectrometer 
(NGIMS) Instrument Ops 

- MAVEN Backup Mission Support Area (bMSA) Backup 
Mission Ops 
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Facility lyild<out 


We held many meetings with the other groups and Science 
Directorate management over the next 6 months 

• We were offered up a large space that had formerly housed the 


Goddard Distributed Active 
Archive Center 

• Our build-out began with the 
collection of excess tables, 
chairs and other computer/ 
printer hardware 

• We also went shopping for 
modular, affordable cubicle 
walls + lockable doors 
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Facility EiuSJd'DUi (2) 


In order to have access to existing pressurized-floor cooling and 
additional UPS-backed power circuits for our racks, we received 
permission to extend our room into an adjacent server datacenter by 
constructing a server cage 


• This saved our five 
projects an 
estimated $230k 
each - over 
$1 million combined 
in additional facility 
build-out costs 
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A “SPOCC" is lom-SPOCC On! 


• Selected Science and Planetary 
Operations Control Center, or 
“SPOCC”, as our name 

- An “unofficial mascot,” the 
SPOCC, was born! 

• Now to define the LADEE Ground 
System Architecture 

- To meet LADEE SOC 
requirements 

- To evolve into a multi-mission 
SPOCC Ground System 
Architecture! 
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We passed LADEE PDR with 
an architecture that could go 
either virtual or physical 

Our biggest hurdle towards 
virtualization was selling it! 

(At the time) Goddard didn’t have 
much of a heritage with 
virtualization on the mission ops 
side, but as a Class D mission 
LADEE was encouraged to 
demonstrate new emergent 
technologies for future missions 
to build upon 
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Option #1 : Physical Servers 


Physical 

Architecture 

Cost 

Quantity 

Total 

Dell R610 Server 4 cores / 
8GB RAM 

$2,800 

26 

$72,800 

Wyse R50L Thin Client 

$450 

12 

$5,400 

24" 1080p HDMI Monitors 

$210 

24 

$5,040 

Windows Server 2008 R2 
License 

$629 

2 

$1,258 

8X8 HDMI Matrix Switcher 

$2,075 

1 

$2,075 







Total Cost: 

$86,573 



For Option #1 (Physical Servers), the total cost was $86,573 for the base set of 
hardware and licenses 

• Net Datacenter Mean Load = 8.1 kW 

• This equates to a net monthly energy usage of 5833 kWh 

At current Maryland commercial energy rates, this would cost the government 
~$6558 / yr, not counting cooling expenses 
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Option #2: WtuaMnd Sentrs 


Virtual 

Architecture 

Cost 

Quantity 

Total 

PowerEdge R710 Server, 2 
sockets / 24 logical cores / 
96GB RAM 

$7,755 

2 

$15,510 

Raid Inc. Xanadu 230 6.5TB 
Direct-Attach SAS Storage 
Array 

$14,818 

1 

$14,818 

Vmware Essentials Plus 
License 

$4,452 

1 

$4,452 

Wyse R50L Thin Client 

$450 

12 

$5,400 

24" 1080p HDMI Monitors 

$210 

24 

$5,040 

Windows Server 2008 R2 
License 

$629 

2 

$1,258 

8X8 HDMI Matrix Switcher 

$2,075 

1 

$2,075 







Total Cost: 

$48,553 


For Option #2 (Virtualized Servers), the total cost was $48,553 for the base set 
of hardware and licenses 

• Net Datacenter Mean Load = 0.9 kW 

• This equates to a net monthly energy usage of 659 kWh 

At current Maryland commercial energy rates, this would cost the government 
~$742 / yr, not counting cooling expenses 
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Virtualization Benefits 






Datacenter Cost! Virtual is better: 

Procurement is 44% cheaper, uses 89% less power! 

Takes up less space in the rack (a half-populated rack instead of 
fully-populated!) 

Provides other benefits: 


LADCE SOC ; NISN Rack t 


I ill 


Our virtual 
system (half) 
rack layout 


- More efficient SysAdmin management 
(snapshots / patching) 

Increased fault tolerance (any single hardware failure would 
only result in minutes of downtime for a system) 

Flexibility in provisioning: Several times during implementation, 
we realized life would be easier if we had an extra Linux 
instance to assign to some task — with Virtualization were 
able to simply provision a new VM (this applies to new 
missions too)! 

Needless to say, it didn’t take too much work to convince our 
stakeholders that virtualization was the most appropriate choice 
for the SPOCC! 
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SPOCC Virtualization Architecture 
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Dsploym^nt EroviroMint / 
IT Security Controls 


• The SPOCC uses a Cobbler PXE boot server / Puppetmaster 
server and Kickstart files to automatically image new Linux 
systems 

• All system configuration and software installation is performed 
and enforced by the Puppet agent (done at install-time and 
then every 30 minutes) 

• Puppet is responsible for implementing all host-specific CIS 
benchmark / NIST 800-53 controls 

• Puppet provides for continuous monitoring and enforcement 
of those controls, software upgrades and allows for 
centralized system-level Configuration Management 
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Thin Client J'Jlsirisiy 3rn an i 


• The SPOCC thin clients are PXE booted into a Linux 
Terminal Services Project (LTSP) 64-bit Enterprise Linux 
6 desktop environment 

• LTSP utilizes a centralized server to distribute thin client 
and kernel images to boot all thin clients 

• Adding new thin clients simply requires updating the 
MAC addresses configuration file 

• Performing patching / configuration updates across all 
thin clients requires simply updating the root image 
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Gj'jISEC 



Architecture 


• The SPOCC utilizes a GMSEC message bus architecture to 
allow applications to communicate via a standard interface 

• GMSEC is used for telemetry event monitoring, system and 
device heartbeat/health information transfer, event filtering / 
analysis and automated notification via email and text message 
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• As of early 2014, four new tenant missions are in various 
phases of planning their integration into the SPOCC 
architecture: 

- DSCOVR Science Operations Center (DSOC) 

- Icesat2 ATLAS Instrument Support Facility (ISF) 

- Magnetospheric Multiscale Mission (MMS) Backup Mission 
Operations Center 

- NICER Payload Operations Center 

• Each is planning on taking advantage of some level of 
virtualization + shared resources 
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Lessons Learned 


• “If you build it, they will come!” 

• It’s definitely worth spending the extra 
time up-front when engineering a 
solution to make it generally applicable 

• Documentation is key-document early, 
and document often! It easily takes 
1/1 0 th the time to document something 
as you are doing vs. after-the-fact... 

• When implementing something new, 
make sure you research industry best 
practices before settling on a single technique or approach! 

• Traditional practices on sparing and string management are not 
applicable the same way in virtualization, it requires a re-evaluation 
of core concepts 
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Backmip Slides 
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SPOCC Facility 
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GS'iJSEC Component Rotes 


GMSEC Tool 

Description 

Features 

CAT 

Criteria Action Table 

Process software component messages 

GEDAT 

GMSEC Environmental 
Diagnostic Analysis Tool 

Visualization of SA data, heartbeats and 
Middleware Health 

GREAT 

Event / log messaging for bus 

Debugging GMSEC; message archive 

ANSR 

Paging system 

Page operations users via outgoing lONet e- 
mail gateway in response to specified events 

SA 

System Agents 

Provide heartbeats for middleware and 
operating system instances, provide rolled- 
up system status to the GMSEC bus 

ActiveMQ 

Open-source Middleware 

Provide reliable connectivity between SOC 
systems that will be generating alerts, CAT 
and ANSR 

GMSEC-API 

GMSEC Application 
Programming interface 

Serve as an interface between different 
software programs 
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